Analysis of Optimization Experiments

The typical practice for analyzing industrial experiments is to identify statistically significant effects with a 5% level of significance and then to optimize the model containing only those effects. In this article, we illustrate the danger in utilizing this approach. We propose methodology using the practical significance level, which is a quantity that a practitioner can easily specify. We also propose utilizing empirical Bayes estimation which gives shrinkage estimates of the effects. Interestingly, the mechanics of statistical testing can be viewed as an approximation to empirical Bayes estimation, but with a significance level in the range of 15-40%. We also establish the connections that our approach has with a less known but intriguing technique proposed by Taguchi, known as the beta coefficient method. A real example and simulations are used to demonstrate the advantages of the proposed methodology.